The present invention relates generally to real-time event processing, and more particularly to methods, apparatus and systems for processing real-time events in applications such as telecommunications and computer networks.
Many event-processing systems have real-time performance requirements that cannot be met by conventional general-purpose database management systems. In telecommunications applications, for example, a variety of adjunct switching services such as debit-based billing, number mapping, call forwarding, and local-number portability involve event processing during the critical call-connection phase of a telephone call. To meet the real-time requirements of the network, the service time for such events generally must not exceed a few milliseconds. However, with conventional database technology, the costs of invoking a structured query language (SQL) operation over a client-server interface, or the costs associated with a single access to secondary storage, can account for hundreds of milliseconds. As a consequence, performance goals on the order of a few milliseconds may already be unattainable even before the costs of the event processing logic are taken into account.
These limitations have led to the use of custom database systems for many high-performance real-time event processing applications. Custom systems are typically tightly coupled to their particular applications, and are tuned to the specific requirements of those applications. These solutions generally work well in practice, and can provide close to optimal performance. However, they also have a number of serious disadvantages. For example, the cost of developing and maintaining custom systems can be high, and generally cannot be amortized across a number of different applications. Moreover, custom systems are frequently inflexible. It can become difficult or even impossible to adapt a custom system to unforeseen or evolving requirements.
A need therefore exists for an improved real-time event processing system which provides the performance benefits of custom database systems, without sacrificing the flexibility and maintainability typically associated with conventional general-purpose database systems.
The invention provides a general-purpose real-time event processing system (EPS) which avoids the problems associated with custom systems. The EPS in an illustrative embodiment meets real-time performance goals through the use of one or more real-time analysis engines (RAEs) operating in conjunction with a main-memory storage manager as its underlying database system. The main-memory storage manager offers transactional access to persistent data, but at the speed of a main-memory system. The EPS also incorporates a recovery model which stores recovery information in order to facilitate roll-back to a recovery point after an RAE failure.
In accordance with the invention, recovery information regarding a recovery point for a given RAE or set of RAEs in an EPS is stored in a memory portion of the main-memory database system of the EPS. Serial tags are assigned to the events, e.g., in a monotonically increasing sequence, in accordance with an arrival order of the events, and the tags and the corresponding events are stored in an event log of the EPS. The recovery information stored in the memory of the main-memory database system includes the serial tag for each event for which processing in the real-time analysis engine has reached a commit operation. After an event is processed and the results of the processing stored in an external database storage element, e.g., a data warehouse or enterprise store, the serial tag of the event is stored in a serial tag log associated with that storage element. In the event of a failure of the given RAE, the serial tags stored in the memory portion of the main-memory database system and the serial tags stored in the serial tag log of the external database storage element are utilized, in conjunction with stored event information from the event log, to implement a roll-back of the RAE to the recovery point.
A real-time EPS in accordance with the invention provides a critical path for event processing that is specifically designed for high performance, while also retaining many desirable features of conventional database systems, including high-level, declarative programming interfaces, and the well-known transactional correctness properties of atomicity, consistency, isolation and durability (ACID). These features of the invention enhance the reliability, robustness, usability and maintainability of the real-time EPS and the applications built thereon.
The invention can be used in conjunction with any event processing application, including, for example, telecommunications, electronic commerce, and Internet service provisioning applications. For example, in a telecommunications application, the invention can be configured to provide a basis for features such as enhanced billing systems, fraud detection and prevention, local-number portability, settlements among service providers and real-time traffic analysis.